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IDENTIFYING COMMON MATHEMATICAL MISCONCEPTIONS 


FROM ACTIONS IN EDUCATIONAL VIDEO GAMES 

Deirdre Kerr 

CRESST/ University of California, Los Angeles 

Abstract 

Educational video games provide an opportunity for students to interact with and explore 
complex representations of academic content and allow for the examination of problem- 
solving strategies and mistakes that can be difficult to capture in more traditional 
environments. However, data from such games are notoriously difficult to analyze. This 
study used a three-step process to examine mistakes students make while playing an 
educational video game about the identification of fractions. First, cluster analysis was 
used to identify common misconceptions in the game. Second, a survey was given to 
determine if the identified in-game misconceptions represented real-world 
misconceptions. Third, a second educational video game was analyzed to determine 
whether the same misconceptions would be identified in both games. Results indicate 
that the in-game misconceptions identified in this study represented real-world 
misconceptions and demonstrate that similar misconceptions can be found in different 
representations. 

Introduction 

Educational video games provide an opportunity to present students with authentic and 
interesting educational tasks (Edelson, Gordin, & Pea, 1999) in an environment where they 
can interact with and explore complex representations of serious academic content (Fisch, 
2005; National Research Council, 2011). Additionally, educational video games record the 
exact learning behavior of students, not just the answers given (Romero & Ventura, 2007). 
This allows for the examination of problem-solving strategies and mistakes that can be 
impossible to capture on a paper-and-pencil test (Merceron & Yacef, 2004; Quellmalz & 
Pellegrino, 2009; Rahkila & Karjalainen, 1999) or through students’ verbal explanations 
(Bejar, 1984). 

The resulting information can be used to provide detailed measures of the extent to 
which players have mastered specific learning goals (National Science and Technology 
Council, 2011) or to support diagnostic claims about students’ learning processes (Leighton 
& Gierl, 2007). Educational games and simulations also have the potential to be used to 
identify the strengths and weaknesses of individual students (Mehrens, 1992), provide 
individualized feedback (Brown, Hinze, & Pellegrino, 2008), guide instruction that is optimal 
for each student (Bejar, 1984; Clark, Nelson, Sengupta, & D’Angelo, 2009; Radatz, 1979), or 
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allow students to track their own progress (Rahkila & Karjalainen, 1999). Additionally, they 
could be used to improve classroom instruction (Merceron & Yacef, 2004) by allowing for 
the identification of common errors or determining the relative effectiveness of different 
pedagogical strategies for different types of students (Romero & Ventura, 2007). 

These possibilities have led the government to call for the research and development of 
educational video games and simulations as platforms to assess the complex skills identified 
in state and national standards (U.S. Department of Education, 2010) and determine the 
effectiveness of different instructional practices (U.S. Department of Education, 2012). 

However, the interpretation of the rich stream of complex data that results from the 
tracking of in-game actions is one of the most serious bottlenecks facing researchers 
examining educational video games and simulations today (Mislevy, Almond, & Lukas, 
2004). The task is so difficult that a government task force recently determined that the single 
biggest challenge to embedding assessment in educational games and simulations is 
determining methods of drawing inferences from log data (National Research Council, 
2011). The process of identifying evidence of student performance in educational video 
games and simulations is incredibly complex due to the sheer number of observable actions 
and the variety of potential relationships each action could have with student performance 
(Frezzo, Behrens, Mislevy, West, & DiCerbo, 2009). Extracting relevant features from the 
noise in the data is crucial in such environments to make analysis computationally tractable 
(Masip, Minguillon, & Mor, 2011). 

In educational video games or simulations, relevant features of student performance 
must be extracted from the log files that are automatically generated by the game or 
simulation as students play (Kim et al., 2008). Though log data are more comprehensive and 
more detailed than most other forms of assessment data, analyzing log data presents a 
number of challenging problems (Garcia, Romero, Ventura, de Castro, & Calders, 2011; 
Mostow et al., 2011) inherent when examining exact learning behaviors in highly 
unstructured environments (Amershi & Conati, 2011). These environments typically include 
thousands of pieces of information for each student (Romero, Gonzalez, Ventura, del Jesus, 
& Herrera, 2009) with no known theory to help identify which information is salient 
(National Research Council, 2011). In addition to the size of the data, the specific 
information stored in the log files is not always easy to interpret (Romero & Ventura, 2007), 
as the responses of individual students are highly context dependent (Rupp, Gushta, Mislevy, 
& Shaffer, 2010), and it can be very difficult to picture how student knowledge, learning, or 
misconceptions manifest themselves at the level of a specific action taken by the student in 
the course of the game. 
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Due to these difficulties, there is currently no systematic approach to extracting 
relevant data from log files (Muehlenbrock, 2005) and the field is still in its infancy 
(Romero, Ventura, Pechenizkiy, & Baker, 2011; Spector & Ross, 2008). 

Related Work 

Some researchers have resorted to hand coding log files to extract the relevant data. 
Trained human raters have been used to extract purposeful sets of actions from game logs 
(Avouris, Komis, Fiotakis, Margaritis, & Voyiatzaki, 2005) and logs of eye-tracking data 
(Conati & Merten, 2007) and to identify student errors in log files from an introductory 
programming environment (Vee, Meyer, & Mannock, 2006). Additionally, Amershi and 
Conati (2011) had raters examine behavior patterns in log files from an exploratory learning 
environment and categorize students as high learners, thoughtful low learners, or 
unthoughtful low learners. 

A number of other studies used basic aggregate information from the log data from 
online learning environments, without examining the specific actions taken by students. The 
aggregate information extracted from the log data were the number of activities completed by 
the student and the amount of time spent in the activity. The number of activities completed 
in the online learning environments Mooclle (Romero et al., 2009) and ActiveMath (Scheuer, 
Miihlenbrock, & Melis, 2007) have been used to predict student grades. The time spent in 
each activity in an online learning environment has been used to detect unusual learning 
behavior (Ueno & Nagaoka, 2002). Combinations of the total time spent in the online 
environment and the number of activities successfully completed have been used to predict 
student success (Muehlenbrock, 2005) and report student progress (Rahkila & Karjalainen, 
1999). 

Other studies focused on summarizations or averages of pre-coded events. One such 
study counted the number of hints students requested and the number of failures they 
experienced to categorize students as hint-driven learners or failure-driven learners 
(Yudelson et al., 2006). A second study counted the number of deaths in each area of the 
game to determine which areas needed improvement (Kim et al., 2008). A third study 
counted the amount of money earned in a management simulation to determine effective or 
ineffective players (Ramnarayan, Strohschneider, & Schaub, 1997). A fourth study counted 
the number of errors, the average economy of motion, and the time it took students to finish a 
laparoscopic surgery simulation to determine performance (Gallagher, Lederman, McGlade, 
Satava, & Smith, 2004). 
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Study Design 


In this report, a three-step process is used to examine mistakes students make while 
playing an educational video game. After the misconceptions were identified, the second step 
of the study was to give a later sample of students a survey about the identified 
misconceptions. The purpose of the survey was to determine whether the identified in-game 
misconceptions reflected real-world misconceptions of fractions and to confirm that the 
interpretation of each misconception was accurate. Finally, log data from a similar game 
were examined. The purpose of examining log data from a second educational video game 
focusing on the same topic was to determine whether the same misconceptions would be 
identified in both contexts and to ascertain whether individual students were identified as 
holding the same misconceptions in both games. 

Identification of Misconceptions 

Study Design 

In the initial step of the study, a small sample of students played a fractions game called 
Save Patch. Each action the students took in the game was logged automatically and 
analyzed using data mining techniques to answer the following research question: 

1. Can mathematical misconceptions be identified solely from actions students take in 
an educational video game? 

Methods 

Save Patch was designed to address four main fractions concepts: the meaning of the 
unit, the meaning of addition as applied to fractions, the meaning of the denominator, and the 
meaning of the numerator (Vendlinski, Delacruz, Buschang, Chung, & Baker, 2010). In order 
to address these concepts, the game area was represented as a line in one-dimensional levels 
and a grid in two-dimensional levels to reinforce the idea that a unit can be represented as 
one whole interval on a number line. Units were represented as gray posts connected by dirt 
paths, with small red posts indicating the fractional pieces the unit was broken into (see 
Figure 1). Students were given ropes in the resource bin labeled Path Options and had to 
break the ropes they were given into the fractional pieces indicated in the level and place the 
correct number of unit fractions (fractions with a numerator of one) on each sign to guide 
their character safely to the cage to unlock the prize. 


4 




Figure 1. Screen shot from Save Patch. 

Successful game play required students to determine the unit size for a given grid as 
well as the size of the fractional pieces making up each unit. The distance the character 
moved was a function of the number and size of ropes placed on each sign, where one whole 
rope represented a whole unit and each whole rope could be easily broken into fractional 
pieces of the desired size by clicking on the arrows next to the rope. Therefore, a successful 
solution to a given level should indicate a solid understanding of fractions. 

To correctly solve the level in Figure 1, the steps of a successful solution would be to 
(a) convert one of the whole unit coils in the Path Options into fourths by clicking on the 
down arrow next to the rope, (b) drag a 1/4 piece of rope onto the beige sign to the right of 
the game character in the upper left corner of the screen, (c) drag another 1/4 piece of rope to 
the second beige sign, (d) drag another 1/4 piece of rope to the third beige sign, (e) drag a 
whole unit rope piece to the gold sign, (f) change the direction of the last sign to down by 
clicking on the arrow beneath the sign, and (g) click on the Go button to submit the answer. 
Other valid solutions to the level include: (1) placing three 1/4 pieces of rope on the first 
beige sign, placing nothing on the other beige signs since having 3/4 on the first sign would 
make the game character walk directly to the gold sign, and placing a whole unit rope on the 
gold sign, or (2) breaking an additional whole unit rope into fourths in the Path Options and 
placing four 1/4 pieces of rope on the gold sign rather than one whole unit piece of rope. 

This design allowed students to demonstrate knowledge of the meaning of the 
denominator of a fraction (by breaking up the whole units into the correct denominator) and 
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the numerator of a fraction (by placing the correct number of unit fractions on each sign). 
Additionally, a number of levels in the game were designed to represent more than one unit, 
allowing students to demonstrate knowledge of the meaning and importance of the whole 
unit. 

To allow students to demonstrate knowledge of addition as applied to fractions, game 
play was constrained so that it was not possible to add two fractions with different 
denominators, rather than allowing the students to make the addition and have the game 
calculate the resulting distance. This means that the game did not allow students to add 1/2 to 
1/3, but instead forced students to convert the 1/2 rope to 3/6 and the 1/3 rope to 2/6 before 
allowing the fractions to be added together. For the same reason, the game did not allow the 
creation of mixed numbers (e.g., 1 1/2), forcing players to convert the whole number portion 
of the mixed number into the appropriate fractional representation (e.g., 2/2) before adding 
the fractional portion to the whole number portion. 

To scaffold students’ understanding of fractions and provide a logical progression 
through the game, Save Patch was broken into six stages. The first stage was designed to 
introduce students to the game mechanics in a mathematical setting they were comfortable 
with, and therefore included only whole numbers. The second stage introduced fractions via 
unit fractions, requiring students to identify the denominator while restricting the numerator 
to one. The third stage combined concepts from the first two stages, with at least one distance 
in each level representing a unit fraction and at least one other distance representing a 
distance equivalent to a whole unit. The fourth stage was similar to the third stage, except 
that the distance representing a whole unit did not start and end on unit markers. Instead, the 
whole unit distance spanned a unit marker (e.g., extending from 1/3 to 4/3). The fifth stage 
dealt with proper fractions (where the numerator was greater than one but smaller than the 
denominator) and was when students were first asked to identify the numerator as well as the 
denominator of a fraction. The sixth stage completed the identification of fractions concepts 
by asking students to identify improper fractions (where the numerator was larger than the 
denominator). 

The sample consisted of 155 students from an urban school district in southern 
California in sixth-grade math, Algebra Readiness, or Algebra 1 courses. These students 
played Save Patch for approximately 40 minutes in their regular math class. The resulting 
game log data were analyzed using fuzzy feature cluster analysis to group in-game actions 
that commonly co-occurred, resulting in the identification of two common mathematical 
misconceptions in the game (details of the process can be found in Kerr & Chung, 2012). 
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Results 


The most common misconception held by students involved a misunderstanding of how 
fractions were partitioned. Students who had partitioning misconceptions could not correctly 
determine the denominator of the fraction. Rather than counting the number of pieces each 
unit was broken into to determine the denominator, these students counted the number of 
dividing marks between pieces. This misconception caused students to consistently identify 
the denominator of a fraction incorrectly. In the example in Figure 2, a student who knew 
how to partition a fraction correctly would count the number of spaces in the first unit, which 
would lead them to identify the denominator as 3. This student would then place three 1/3 
pieces on the first sign, one 1/3 piece on the second sign, and one 1/3 piece on the third sign, 
resulting in a successful attempt to reach the prize. 

However, students who held misconceptions about partitioning would count the 
number of small red posts in the first whole unit, rather than the number of spaces. In the 
example in Figure 2, this would lead them to identify the denominator as 2. This student 
would then place three 1/2 pieces on the first sign, one 1/2 piece on the second sign, and one 
1/2 piece on the third sign, resulting in an unsuccessful attempt to reach the prize. 



Figure 2. Partitioning errors in Save Patch. 


This misconception results in a successful solution whenever the fractional 
representation is a circle. In circular representations of fractions, such as slices of pizza or 
pie, the number of pieces and the number of lines between the pieces are generally the same. 
You cut a line into three pieces by making two cuts, but you cut a circle into three pieces by 
making three cuts. Save Patch did not reinforce this misconception because there were no 
circular representations of fractions in the game. 

The other common misconception held by students involved a misunderstanding about 
the unit. Students who made this error could not correctly determine the number of units 


7 




being represented. Rather than using the labeling and/or visual cues provided in the 
representation (such as large dividing marks) to determine the number of units, these students 
always assumed that the entire representation was one unit across. In the example in Figure 
3, a student who knew how to identify the unit correctly would determine that there were two 
units represented and count the number of spaces in only the first unit to determine the 
denominator, which would lead them to identify the denominator as 3. This student would 
then place three 1/3 pieces on the first sign, one 1/3 piece on the second sign, and one 1/3 
piece on the third sign, resulting in a successful attempt to reach the prize. 

However, students who held misconceptions about unitizing would assume that there 
was only one unit represented and would, therefore, count the number of spaces in the entire 
representation (rather than only the number of spaces in the first unit). In the example in 
Figure 3, this would lead them to identify the denominator as 6. This student would then 
place three 1/6 pieces on the first sign, one 1/6 piece on the second sign, and one 1/6 piece on 
the third sign, resulting in an unsuccessful attempt to reach the prize. 



Figure 3. Unitizing errors in Save Patch. 


This misconception would result in a successful solution whenever the representation 
consisted of only one unit. In such cases, their assumption that the representation was always 
one unit across would be correct. Save Patch may have reinforced this misconception 
because approximately a third of the game levels represented a single unit. 

The cluster analysis also identified two game -related errors specific to the number line 
representation in Save Patch. The most common game -related error was for students to avoid 
math by using all the resources in the order in which they were provided, rather than 
attempting to calculate the denominator represented in the level. For example, if students 
were given one half, one third, and one fourth in the Path Options, students who made this 
error would place the one half rope on the first sign on the grid, the one third rope on the 


second sign on the grid, and the one fourth rope on the third sign on the grid. The other 
game-related error was a directional error involving misuse of the arrows beneath each sign. 
Students who made directional errors failed to change the direction of the arrow before 
submitting their answer in levels where a directional change was required, or changed the 
direction so that the puppet walked the wrong way on the grid and failed to reach the goal 
(despite having the correct values on each sign). 

Gathering Evidence to Support Inferences 

Study Design 

In the second step of the study, a subsequent sample of students were given a short 
survey after game play to answer the following research questions: 

1. Do the identified mathematical misconceptions reflect real-world misconceptions of 
fractions, rather than simply being due to oddities in the game medium? 

2. Are the interpretations of how these misconceptions occur accurate? 

Methods 

This survey was given to 484 sixth-grade students in 22 urban and suburban schools 
who had previously played Save Patch for four days in their regular math class. The survey 
presented students with two different representations of 4/3 (shown in Figure 4). One 
representation showed a level from the game where the prize was located at 4/3, and the 
other representation showed a number line with a question mark located at 4/3. Students were 
asked to (a) identify the location of the prize and explain how they got their answer, (b) 
identify the location of the question mark and explain how they got their answer, and (c) state 
whether they thought the two representations were the same or different and explain why 
they thought so. 



Figure 4. Game and number line representations of the same problem (identifying 4/3). 


Results 

As can be seen in Figure 5, a majority of the students (76%) recognized that the two 
questions represented the same problem. Additionally, most of the students who recognized 
that the questions were the same gave a conceptual explanation of the similarity (e.g., 
“because they are two different pictures of the same number line” or “because the cage is in 
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the same place as the question mark”) while far fewer students based their explanation on the 
similarity of their answers. In fact, 35% of students who said that the questions were the 
same actually provided different answers to Question A and Question B. These results 
demonstrate that most students view the game medium as equivalent to a real-world number 
line. Additionally, 57% of students who made errors reflecting one of the identified 
misconceptions on the game question made the same error on the number line question, 
providing some evidence that the identified misconceptions reflect real-world 
misconceptions of fractions. 
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Figure 5. Responses to whether the questions were the same broken out by 
reason given. 


More than half of the students (59%) made errors on either Question A or Question B. 
While many of the errors involved neither unitizing nor partitioning misconceptions, 38% of 
errors were partitioning errors and 19% were unitizing errors. In total, 93 of the 484 students 
in the study made partitioning errors, 38 made unitizing errors, and 16 made errors involving 
both unitizing and partitioning. Figure 6 shows the percentage of students making each error 
who provided explanations for their answers that was consistent with the interpretation of the 
misconception. 
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Figure 6. Percentage of each error type where explanation matched expectations. 


Most explanations were too vague to be interpreted (e.g., “I counted” or “Because”). 
However, 39% of students who made a partitioning error explicitly stated that they counted 
lines to determine the denominator and 52% of students who made a unitizing error explicitly 
stated that they counted “all the way to the end” to determine the denominator. Of the 
students who made both unitizing and partitioning errors, 25% stated that they counted lines 
all the way to the end, an additional 19% stated that they counted lines (but did not mention 
counting all the way to the end) and 19% stated that they counted all the way to the end (but 
did not mention counting lines). Most importantly, not a single student stated that they 
counted something besides lines to determine the denominator or stated that they stopped 
somewhere besides the end, indicating that the interpretations of how these misconceptions 
occur are largely accurate. 

Identifying Misconceptions in Multiple Contexts 

Study Design 

In the third step of the study, students played a different game about identifying 
fractions before playing Save Patch and took a paper-and-pencil test before and after game 
play. This study investigated the following research questions: 
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1. Can the same mathematical misconceptions be identified in other games that 
address the same topic? 

2. Are individual students identified as holding the same misconceptions in both 
games? 

Methods 

The sample consisted of 854 sixth-grade students from 20 urban and suburban schools. 
These students participated in the study for 12 non-consecutive days in their regular math 
classes, for approximately 40 minutes each day. On the first day, students took a pretest 
measuring their prior knowledge of fractions. On Days 2 through 3, students played Wiki 
Jones , an educational game designed to remediate the identification of fractions issues 
identified in Save Patch. On Days 4 through 7, students played Save Patch. On Days 8 
through 11, students played other educational games not related to the identification of 
fractions, and on Day 12 students took a posttest measuring their understanding of fractions. 

Wiki Jones was designed to address three of the four main fractions concepts addressed 
in Save Patch, the meaning of the unit, the meaning of the denominator, and the meaning of 
the numerator. In order to address these concepts, the game area was represented as a number 
line superimposed over images of objects in the background (see Figure 7). In this game, 
students are detectives on the trail of bacon thieves. Depending on the prompt given, students 
would have to identify each whole unit correctly, divide each unit into the correct number of 
pieces, or locate the correct position on the number line to advance the story. 



Figure 7. Screen shot from Wiki Jones. 
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Students could choose which action to take by clicking on the corresponding oval 
button in the upper right hand comer of the screen. The Whole Unit button activated a 
bagging machine that students could use to identify individual units on the number line. In 
the level in Figure 7, students could select the Whole Unit machine and then drag the bag 
from 0 to 1 and then from 1 to 2 to identify the two units shown on the number line. The 
Divide machine activated a laser that would cut the image into smaller pieces. In the level in 
Figure 7, students are being asked to select the Divide machine and draw three equally 
spaced lines across the bacon between the 0 and the 1 and three additional equally spaced 
lines across the bacon between the 1 and the 2. The Locate button (not shown in this level) 
would activate a claw machine that would grab whatever was located at the position 
indicated. If asked to locate an item at 6/4, students would select the Locate machine and 
then click on the 6/4 position on the number line. 

This design allowed students to demonstrate knowledge of the meaning and importance 
of the whole unit (by breaking up a number line into whole units), the meaning of the 
denominator of a fraction (by breaking up the whole units into the correct denominator) and 
the numerator of a fraction (by identifying the location on the number line of the value 
indicated). Because successful game play required students to determine the unit size for a 
given number line, the size of the fractional pieces making up each unit, and the location on 
the number line of a specified value, a successful solution to a given level should indicate a 
solid understanding of fractions. 

To scaffold students’ understanding of fractions and provide a logical progression 
through the game, Wiki Jones was broken into five stages. The first stage addressed only the 
meaning of the whole unit by requiring students to identify the number of whole units being 
represented. The second stage built on the first stage by introducing the meaning of the 
denominator, requiring students to break each unit into the desired number of pieces. The 
third stage built on the previous stages by introducing the meaning of the numerator, 
requiring students to identify the location of a given fraction on a number line already 
divided into the appropriate number of pieces. The fourth stage combined skills from the 
previous stages by requiring students to first break each unit into the desired number of 
pieces before identifying the location of a given fraction. The final stage reversed the process 
by asking students to write down the fractional value of the indicated location on the number 
line. 

The resulting game log data were analyzed using fuzzy feature cluster analysis in the 
same process as applied to Save Patch in Kerr and Chung (2012). While the individual 
actions recorded in the log data from Wiki Jones differ significantly from the individual 
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actions recorded in the log data from Save Patch, the cluster analysis should identify the 
same misconceptions in both games, given that both games address the same mathematical 
content. 

To determine whether there was a relationship between the errors made in the two 
games, correlations between the number of partitioning errors on the pretest, the number of 
partitioning errors made in Wiki Jones, the number of partitioning errors made in Save Patch, 
and the number of partitioning errors made on the posttest were calculated. Similar 
correlations were calculated for unitizing errors. 

Results 

As in Save Patch, the most common misconception held by students in Wiki Jones 
involved a misunderstanding of how fractions were partitioned. Students who had 
partitioning misconceptions could not correctly divide a unit into the required number of 
pieces. Rather than drawing one line less than the number of pieces required (e.g., drawing 
three lines to make four pieces), these students drew the same number of lines as the number 
of pieces required (e.g., drawing four lines in an attempt to make four pieces). The 
misconception caused students to consistently divide units into the wrong fractional amounts. 
In the example in Figure 8, a student who knew how to partition correctly would divide each 
unit into four pieces by drawing three evenly spaced lines inside each unit. Students who held 
misconceptions about partitioning would attempt to divide each unit into four pieces by 
drawing four evenly spaced lines inside each unit, resulting in five pieces rather than four. 


Correct 


Partitioning Error 



Figure 8. Partitioning errors in Wiki Jones. 


As in Save Patch, the other common mathematical misconception held by students in 
Wiki Jones involved a misunderstanding about the unit. Students who had unitizing 
misconceptions could not correctly determine the number of units being represented. Rather 
than using the labeling and/or visual clues provided in the representation to determine the 
number of units, these students always assumed that the entire representation was one unit 
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across. In the example in Figure 9, a student who knew how to identify the unit correctly 
would determine that there were two units represented and draw three lines in the first unit to 
break it into four pieces. Students who held misconceptions about unitizing would assume 
that there was only one unit represented (despite the fact that both units are clearly labeled in 
the representation) and would, therefore, draw three lines evenly spaced between the 
beginning and the end of the represented number line. In the example in Figure 9, this would 
result in the student dividing each unit into two pieces rather than the intended four pieces. 



While the cluster analysis process resulted in the identification of the same 
mathematical errors in both games, entirely different game -related errors were identified. 
Because Wiki Jones did not include directional indicators in any of its representations, it was 
not possible for students to make directional errors in this game. Nor was it possible for 
students to avoid math by using all of the resources in the order in which they were given, 
because Wiki Jones does provide students with a list of available resources. Instead, students 
in Wiki Jones made other game-related errors specific to the game’s representation of 
fractions. 

The game-related errors in Wiki Jones involved accidentally dropping two in-game 
tools: the wrapping tool and the cutting tool. If students dropped the wrapping tool by 
accidentally letting go of the cursor, this error resulted in the identification of a unit that 
began and ended at the same point (and was invisible to students, so it could not be corrected, 
but would still result in an error when students submitted their answer). If a student dropped 
the cutting tool before the line went all the way through the bacon, the game would not make 
the desired cut. Instead, the line would disappear and the student would have to draw it again. 

To examine the relationship between misconceptions in Wiki Jones and misconceptions 
in Save Patch, correlations were run between the number of unitizing errors made in each 
game and the number of unitizing errors made on the pretest and posttest (see Table 1). All 
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correlations were significant at p < .001, indicating that there is a significant relationship 
between unitizing errors made in all four environments. However, the strength of the 
relationships varied. The relationship between unitizing errors in Wiki Jones and in Save 
Patch was not as strong as the relationship between pretest unitizing errors and posttest 
unitizing errors ip < .00 1 ). 

Table 1 

Correlations Between In-Game Unitizing Errors and Pretest and 

Posttest Unitizing Errors 


Unitizing errors 


Unitizing errors 

Pretest 

Wiki Jones 

Save Patch Posttest 

Pretest 

— 



Wiki Jones 

.163 

— 


Save Patch 

.229 

.137 

— 

Posttest 

.422 

.120 

.262 


Note. All correlations significant at p < .001. 


Similar correlations were run for partitioning errors (see Table 2). All correlations were 
significant at p < .001, indicating that there is also a significant relationship between 
partitioning errors made in all four environments. All relationships for partitioning errors 
were stronger than their equivalent relationships for unitizing errors. However, as with 
unitizing errors, the strength of the relationships varied. The relationship between 
partitioning errors in Wiki Jones and in Save Patch was not as strong as the relationship 
between pretest partitioning errors and posttest partitioning errors ip < .001). 

Table 2 

Correlations Between In-Game Partitioning Errors and Pretest and 

Posttest Partitioning Errors 


Partitioning errors 


Partitioning errors 

Pretest 

Wiki Jones 

Save Patch Posttest 

Pretest 

— 



Wiki Jones 

.258 

— 


Save Patch 

.312 

.293 

— 

Posttest 

.494 

.229 

.333 


Note. All correlations significant at p < .001. 
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Discussion 


This study used a three-step process to examine mistakes students made while playing 
an educational video game. First, in-game mathematical misconceptions were identified 
using cluster analysis. Then, a survey was given to determine whether the identified in-game 
misconceptions reflected real-world misconception of fractions and to confirm that the 
interpretation of each misconception was accurate. Finally, a similar game was analyzed to 
determine whether the same misconceptions would be identified in both contexts and to 
ascertain whether individual students were identified as holding the same misconceptions in 
both games. 

In the first part of the study, the cluster analysis successfully identified two in-game 
mathematical misconceptions in Save Patch : misconceptions involving partitioning a unit 
into the correct number of fractional pieces and misconceptions involving identifying the 
unit. The cluster analysis also identified two game-related errors specific to the in-game 
representation of fractions. The findings of this part of the study indicate that cluster analysis 
is a promising method of identifying both misconceptions about the content being taught and 
specific game -related errors by examining the actions students take while trying to solve 
problems in an educational video game. 

In the second part of the study, most students stated that the in-game representation and 
the standard number line representation were the same question, and a majority of students 
who made an error involving either identifying or partitioning a unit on the in-game 
representation made the same error on the number line representation. Additionally, student 
explanations of how they arrived at their answers corresponded to the identified 
misconceptions about identifying and partitioning a unit. The results of this part of the study 
indicate that the cluster analysis process identified real-world mathematical misconceptions 
that accurately represented student thought processes, providing initial evidence that cluster 
analysis is a valid method of identifying misconceptions from in-game actions. 

In the third part of the study, the cluster analysis successfully identified the same 
mathematical misconceptions in a second game, called Wiki Jones. While the mathematical 
misconceptions were the same in both games, the cluster analysis identified different game- 
related errors in Wiki Jones than in Save Patch , indicating that the cluster analysis process 
can be used to differentiate between representation-specific errors and underlying 
mathematical misconceptions. Correlations between occurrences of each specific 
mathematical misconception in each game indicate that there is a significant relationship 
between errors across environments, but the relationship between in-game errors was not as 
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strong as the relationship between pretest and posttest errors. This effect may arise because 
the games provide feedback when errors are made (both explicitly in text form and implicitly 
when the desired effect does not occur), which may influence students to change their 
behavior either temporarily or for the long term. 

In sum, the results of this study indicate that the analysis of student actions in 
educational video games can provide valuable information about underlying misconceptions 
that reflect students’ real-world beliefs which can be prohibitively difficult to capture through 
standard assessments. Extracting this information from in-game actions is the first step in 
addressing the serious challenges inherent in embedding assessment in educational games, 
and demonstrates that salient information about student performance can be extracted from 
game log files. 

Additionally, this study provides information that could be valuable in the classroom. A 
short paper-and-pencil diagnostic test to identify the presence of misconceptions involving 
identifying or partitioning a unit has been developed from the information provided by this 
study (see Appendix A). The scoring rubric for use with the test is provided in Appendix B. 
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Appendix A 

Diagnostic Fractions Assessment 
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Name: 


Diagnostic Fractions Assessment 

1 . Examine the number line below. 


0 1 
a. Where is A located on the number line above? Write your answer as a fraction. 
Answer: 

2. Examine the number line below. 


OR 2 

a. Where is A located on the number line above? Write your answer as a fraction. 

Answer: 

b. Where is B located on the number line above? Write your answer as a fraction. 

Answer: 


3. Examine the number line below. 




2 


a. What is the distance between A and B on the number line above? Write your answer as a 
fraction. 


Answer: 
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b. Where is B located on the number line above? Write your answer as a fraction. 


Answer: 
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Scoring Rubric 


25 



Scoring Rubric 


Use the information in the Using the Scoring Rubric section on the following page to determine the category (know, partitioning, or 
unitizing) for each student’s answer to the questions in the Diagnostic Fractions Assessment. The labels above each question identify 
the specific content addressed by each portion of each item. These labels correspond to the stages for Save Patch. 


Proper Fractions More Than One Unit Identifying a Whole Wholes Across a Unit Improper Fractions 
Student Name (Question la) (Question 2a) Unit (Question 2b) Bar (Question 3a) (Question 3b) 

□ know 

□ know 

□ know 

□ know 

□ know 

□ partitioning 

□ partitioning 

□ partitioning 

□ partitioning 

□ partitioning 

□ unitizing 

□ unitizing 

□ unitizing 

□ unitizing 

□ unitizing 

□ know 

□ know 

□ know 

□ know 

□ know 

□ partitioning 

□ partitioning 

□ partitioning 

□ partitioning 

□ partitioning 

□ unitizing 

□ unitizing 

□ unitizing 

□ unitizing 

□ unitizing 

□ know 

□ know 

□ know 

□ know 

□ know 

□ partitioning 

□ partitioning 

□ partitioning 

□ partitioning 

□ partitioning 

□ unitizing 

□ unitizing 

□ unitizing 

□ unitizing 

□ unitizing 

□ know 

□ know 

□ know 

□ know 

□ know 

□ partitioning 

□ partitioning 

□ partitioning 

□ partitioning 

□ partitioning 

□ unitizing 

□ unitizing 

□ unitizing 

□ unitizing 

□ unitizing 

□ know 

□ know 

□ know 

□ know 

□ know 

□ partitioning 

□ partitioning 

□ partitioning 

□ partitioning 

□ partitioning 

□ unitizing 

□ unitizing 

□ unitizing 

□ unitizing 

□ unitizing 

□ know 

□ know 

□ know 

□ know 

□ know 

□ partitioning 

□ partitioning 

□ partitioning 

□ partitioning 

□ partitioning 

□ unitizing 

□ unitizing 

□ unitizing 

□ unitizing 

□ unitizing 

□ know 

□ know 

□ know 

□ know 

□ know 

□ partitioning 

□ partitioning 

□ partitioning 

□ partitioning 

□ partitioning 

□ unitizing 

□ unitizing 

□ unitizing 

□ unitizing 

□ unitizing 

□ know 

□ know 

□ know 

□ know 

□ know 

□ partitioning 

□ partitioning 

□ partitioning 

□ partitioning 

□ partitioning 

□ unitizing 

□ unitizing 

□ unitizing 

□ unitizing 

□ unitizing 
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Using the Scoring Rubric 

Question la: If a student answers 3/4, check the “know” box for question la. 

If a student answers 3/3 or 3/5, check the “partitioning” box for question la. 

Question 2a: If a student answers 1/3, check the “know” box for question 2a. 

If a student answers 1/2 or 1/4, check the “partitioning” box for question 2a. 

If a student answers 1/6, check the “unitizing” box for question 2a. 

Question 2b: If a student answers 1/1 or 3/3, check the “know” box for question 2b. 

If a student answers 3/2 or 3/4, check the “partitioning” box for question 2b. 

If a student answers 3/6, check the “unitizing” box for question 2b. 

Question 3a: If a student answers 1/1 or 4/4, check the “know” box for question 3a. 

If a student answers 4/3 or 4/5, check the “partitioning” box for question 3a. 

If a student answers 4/8 or 1/2, check the “unitizing” box for question 3a. 

Question 3b: If a student answers 6/4 or 3/2, check the “know” box for question 3b. 

If a student answers 6/3, 2/1 or 6/5, check the “partitioning” box for question 3b. 
If a student answers 6/8 or 3/4, check the “unitizing” box for question 3b. 


Instructional Ramifications 

Partitioning: If your students are making a lot of partitioning errors, you may want to use non- 
circular representations of fractions (such as number lines or brownie trays) so that 
the number of lines and the number of spaces are not the same, and explicitly 
explain that the denominator is the number of parts, not the number of spaces. 

Unitizing: If your students are making a lot of unitizing errors (or knew question la but not 

question 2a), you may want to provide representations that include more than one 
unit (e.g., a number line or more than one pizza or tray of brownies) and provide 
explicit instructions on how to determine the number of units represented. 

Question 2b: If most of your students did not know question 2b, you may want to explain that 

fractions are also representations of division and show that (for example) 4/4 is the 
same as 1 because 4 divided by 4 is one. 

Question 3a: If your students knew question 2b but not question 3a, you may want to have 

students find distances equivalent to 1 that start and end between whole numbers, 
and explain it as a subtraction problem (e.g., 6/4 - 2/4 = 4/4 = 1/1 = 1). 

Question 3b: If most of your students did not know 3b, you may want to provide some 

representations of improper fractions and talk about why you would want to keep 
them as improper rather than converting to mixed numbers (e.g., it is easier to add 
them to another fraction if they stay in fractional form). 
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